It's important to specify the scaling_factor so that we know how the value should be interpreted.

Can we document how this cpu percentage works on a server with multiple CPU cores.
Example, If I have a server with 12 CPU cores, will this "CPU percentage" max to 1200% or to 100%?

I believe the way that the endpoint team has done this in the past is the 12 CPU cores has a max of 1200.0123... right @ferullo ?

Thanks for the input! This host.cpu.pct will be the normalized value ranges from 0 to 1 only. If the server has 12 CPU cores, then this normalized value will be total cpu percentage/12.

Would that make scaling_factor: 100?

This would be the first time a scaled_float is defined in ECS, and we'll need to add support for the defining scaling_factor in the schema. I don't see any issue with the addition - just noting that dependency.

@ebeahan I was thinking to keep scaleing_factor as default value. So value 0.12 for example will be stored into ES as 0.12 itself. Maybe I should just use float here instead? Thanks!

If the server has 12 CPU cores, then this normalized value will be total cpu percentage/12.

Unless there's disagreement on this (I'm fine with it), let's update the definition in the RFC to make this a bit more clear. I'm not sure the current phrasing "normalized by by the number of cores" makes it clear enough.

Could be as simple as adding an example.

Example: For a two core host, if one core is at 100% and the other is at zero, the value should be 0.5.

Or we could make it more literal and say "if there's 12 cores, this should be the average of the 12 cores, between 0 and 1"

On the scaling_factor, it's a required parameter for this datatype, there's no default value here.

Note that the source doc is not affected by it, and would always contain the full float value.

A scaling factor of 100 would mean 0.123 (for 12.3%) would only store 0.12 in the index, for aggregations & so on. If we want to aggregate on full digit percents, a scaling_factor of 100 is appropriate.

@kaiyan-sheng or @cyrille-leclerc is 100 appropriate for this use case? Who can chime in on how observability intends to query this field?

PUT sfloat-1
{ "mappings" : { "properties": {
  "pct" : { "type" : "scaled_float", "scaling_factor": 100 }
}}}
PUT sfloat-1/_doc/1
{ "pct": 0.12 }
PUT sfloat-1/_doc/2
{ "pct": 0.1234 }


PUT sfloat-2
{ "mappings" : { "properties": {
  "pct" : { "type" : "scaled_float", "scaling_factor": 10000 }
}}}
PUT sfloat-2/_doc/1
{ "pct": 0.12 }
PUT sfloat-2/_doc/2
{ "pct": 0.1234 }

GET sfloat-*/_search
GET sfloat-*/_search
{ "size": 0, "aggs" : {
    "percentages" : { "terms" : { "field" : "pct" }
}}}

@webmat Thank you for the explanation!! I think I would prefer a scale_factor of 1000 in this case so for CPU usage, we are keeping one decimal when displaying it in a percentage format. For example, I would like to store 0.1234 to be 0.123 in the index so it will show as 12.3% in percentage. But if this is a concern with saving space for storing these values, scale_factor of 100 is also fine for me. @exekias WDYT?

We can try to do some testing and compare sizes, 100 has always looked a bit limiting, some more precision would be good

In the observer namespaces, ECS uses "ingress" and "egress". For consistency we might want to consider uses those terms.

Can we document the nature of this counter. Is it a monotonic counter that is sometimes reset (e.g. system restart)?

@cyrille-leclerc These values will actually be gauges. For example, host.network.in.bytes will be the total bytes received aggregated among all network interfaces in 10 seconds. Next collection period, this value will change, might go up or down depends on the network usage of the next 10 seconds.

++ on aligning on ingress and egress

thanks @kaiyan-sheng I was used to ever increasing counter this kind of metric but I don't know what's the state of the art today.
One benefit of ever increasing counters being that the collector can miss some collects without losing data.

I agree with monotonic counters being superior here. Capturing rates may make it easier to render the data, however rates are lossy.

As Cyrille pointed out, counters are more resilient to an agent missing a beat (pun intended 😄). But counters are also superior for data rollups. Rolling up initial /10s metrics to /5m or hourly percentiles with a counter is trivial. With rates it's not so easy, as the basic piece of data is already an average. Not sure if we can do better in the Elastic Stack than the tool I was using in the past, though...

I'm not opposed to capturing rates if they're dramatically easier to work with in general, but I would like them to be accompanied with the counter backing them.

WDYT?

I think we're confusing two different things here.
The main purpose for this set of host metrics is to align the different resource providers, such as public cloud infrastructures, physical hosts, VMware clusters, other hypervisors, i.e. bring hosts/VMs to a common denominator. The purpose is not to capture high fidelity system metrics because some providers (AWS, GCP, Azure, etc.) simply don't provide these.
Granted, the Metricbeat system module can do much more than just the proposed CPU (that includes raw counters) metrics and should be used wherever possible. 
By making these metrics part of ECS we want improve user experience, i.e. we can always show basic health information about any hosts anywhere, and recommend users to enable the system module for enhanced details.

@webmat @cyrille-leclerc Agree on the benefit of ever-increasing counters. We will definitely keep all the counters as they are right now from the system module. We only added extra calculation using these counters to get gauges in system module to match whatever we get from other resource providers such as AWS, Azure, and GCP. Unfortunately, they don't provide ever-increasing counters in their monitoring metrics as @sorantis mentioned above.

Also in the UI side, it's definitely easier to have metrics as gauges than counters.

I believe the way that the endpoint team has done this in the past is the 12 CPU cores has a max of 1200.0123... right @ferullo ?

If the server has 12 CPU cores, then this normalized value will be total cpu percentage/12.

Unless there's disagreement on this (I'm fine with it), let's update the definition in the RFC to make this a bit more clear. I'm not sure the current phrasing "normalized by by the number of cores" makes it clear enough.

Could be as simple as adding an example.

Example: For a two core host, if one core is at 100% and the other is at zero, the value should be 0.5.

Or we could make it more literal and say "if there's 12 cores, this should be the average of the 12 cores, between 0 and 1"

I agree with monotonic counters being superior here. Capturing rates may make it easier to render the data, however rates are lossy.

As Cyrille pointed out, counters are more resilient to an agent missing a beat (pun intended 😄). But counters are also superior for data rollups. Rolling up initial /10s metrics to /5m or hourly percentiles with a counter is trivial. With rates it's not so easy, as the basic piece of data is already an average. Not sure if we can do better in the Elastic Stack than the tool I was using in the past, though...

I'm not opposed to capturing rates if they're dramatically easier to work with in general, but I would like them to be accompanied with the counter backing them.

WDYT?

I'm not sure if the following is applicable. But would we eventually want to capture the same metrics for containers as well?

If that's the case, perhaps we should consider defining this set of metrics independently, and make them nestable both under "host" and "container".

Good point! We haven't got to container in inventory schema work yet. @exekias Can we potentially treat containers as it's own VMs and report the same set of host metrics?

On the scaling_factor, it's a required parameter for this datatype, there's no default value here.

Note that the source doc is not affected by it, and would always contain the full float value.

A scaling factor of 100 would mean 0.123 (for 12.3%) would only store 0.12 in the index, for aggregations & so on. If we want to aggregate on full digit percents, a scaling_factor of 100 is appropriate.

@kaiyan-sheng or @cyrille-leclerc is 100 appropriate for this use case? Who can chime in on how observability intends to query this field?

PUT sfloat-1
{ "mappings" : { "properties": {
  "pct" : { "type" : "scaled_float", "scaling_factor": 100 }
}}}
PUT sfloat-1/_doc/1
{ "pct": 0.12 }
PUT sfloat-1/_doc/2
{ "pct": 0.1234 }


PUT sfloat-2
{ "mappings" : { "properties": {
  "pct" : { "type" : "scaled_float", "scaling_factor": 10000 }
}}}
PUT sfloat-2/_doc/1
{ "pct": 0.12 }
PUT sfloat-2/_doc/2
{ "pct": 0.1234 }

GET sfloat-*/_search
GET sfloat-*/_search
{ "size": 0, "aggs" : {
    "percentages" : { "terms" : { "field" : "pct" }
}}}

We have in ES now _meta which also support byte as unit: elastic/elasticsearch#61941 Even though it is unfortunate that in older metrics we still have the postfix of .bytes, I think all new values we add we should skip the unit postfix and use _meta instead.

@webmat ECS should add support for _meta.

This is a good point, and I think we should avoid postfixes just for the sake of making the unit explicit. In the case of system.network.out object (this metric comes from it) this is what we have today:

    "system": {
        "network": {
            "in": {
                "bytes": 37904869172,
                "dropped": 32,
                "errors": 0,
                "packets": 32143403
            }
        } 
     }

I wonder how we should differentiate all the different metrics there, specially between bytes and packets

Ha, interesting case. I'm tempted to say in this context the unit name might make sense. In any case, we should still add the info to _meta as this is what should be used by Kibana.

Alternative bytes could be called data but not sure if that is better.

Good point! Thank you @exekias and @ruflin. In this case, I still feel like bytes makes more sense in the metric name instead of data(too broad). bytes in this case, comparing with packets, is identifying the type of this value instead of only being the unit postfix.